Markov Process 

A stochastic process is a function whose values are random 
variables 

The classification of a random process depends on different 
quantities 

- state space 

- index (time) parameter 

- statistical dependencies among the random variables X(t) for different 
values of the index parameter t. 
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Markov Process 

State Space 

- the set of possible values (states) that X(t) might take on. 

- if there are finite states => discrete-state process or chain 

- if there is a continuous interval => continuous process 

Index (Time) Parameter 

- if the times at which changes may take place are finite or countable, 
then we say we have a discrete-(time) parameter process. 

- if the changes may occur anywhere within a finite or infinite interval 
on the time axis, then we say we have a continuous-parameter 
process. 
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Markov Process 

In 1907 A.A. Markov published a paper in which he defined 
and investigated the properties of what are now known as 
Markov processes. 

A Markov process with a discrete state space is referred to as 
a Markov Chain 

A set of random variables forms a Markov chain if the 
probability that the next state is S<n+i> depends only on the 
current state Sw, and not on any previous states 
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Markov Process 


States must be 

- mutually exclusive 

- collectively exhaustive 

Let Pi t) = Probability of being in state S t at time t. 


= 1 



Markov Properties 

- future state prob. depends only on current state 
independent of time in state 
path to state 
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Markov Process 


Assume exponential failure law with failure rate X. 

Probability that system failed at t + At, given that is was 
working at time t is given by 
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Markov Process 
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Markov Process 

Accessibility 

- state S t is accessible from state S- if there is a sequence of transitions 
from 5, to S,. 

Recurrent State 

- state Sj is called recurrent, if S t can be returned to from any state 
which is accessible from S t in one step, i.e. from all immediate 
neighbor states. 

Non Recurrent 

- if there exists at least one neighbor with no return path. 
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Markov Process 


Classes of States 

- set of states (recurrent) s.t. any state in the class is reachable from any 
other state in the class. 

- note: 2 classes must be disjoint, since a co mm on state would imply 
that states from both classes are accessible to each other. 


Absorbing State 

- a state S t is absorbing iff 
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